Introduction to tidyverse/R

Course Setup

install.packages("tidyverse")

library(tidyverse)

-- Attaching packages --------------------------------------- tidyverse 1.2.1 --
v ggplot2 2.2.1     v purrr   0.2.4
v tibble  1.3.4     v dplyr   0.7.4
v tidyr   0.7.2     v stringr 1.2.0
v readr   1.1.1     v forcats 0.2.0
-- Conflicts ------------------------------------------ tidyverse_conflicts() --
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()

Explore Data

First ggplot

ggplot(data = midwest) +
  geom_point(mapping = aes(x = popdensity, y = percollege))

Equivalent Code

ggplot(midwest) +
  geom_point(aes(x = popdensity, y = percollege))

Your Turn

1. Try plotting popdensity by state.
2. Try plotting county by state.
   • Does this plot work?
3. Bonus: Try just using the ggplot(data = midwest) from above.
   • What do you get?
   • Does this make sense?

Add Aesthetics

ggplot(midwest) +
  geom_point(aes(x = popdensity, y = percollege, color = state))

Global Aesthetics

ggplot(midwest) +
  geom_point(aes(x = popdensity, y = percollege), color = 'pink')

Your Turn

1. Instead of using colors, make the shape of the points different for each state.
2. Instead of color, use alpha instead.
   • What does this do to the plot?
3. Try the following command: colors().
   • Try a few colors to find your favorite.
4. What happens if you use the following code:

ggplot(midwest) + 
  geom_point(aes(x = popdensity, y = percollege, color = 'green'))

Additional Geoms

ggplot(midwest) +
  geom_smooth(aes(x = popdensity, y = percollege))

Add more Aesthetics

ggplot(midwest) +
  geom_smooth(aes(x = popdensity, y = percollege, linetype = state), 
              se = FALSE)

Your Turn

1. It is possible to combine geoms, which we will do next, but try it first. Try to recreate this plot.

Layered ggplot

ggplot(midwest) +
  geom_point(aes(x = popdensity, y = percollege, color = state)) +
  geom_smooth(aes(x = popdensity, y = percollege, color = state), 
              se = FALSE)

Remove duplicate aesthetics

ggplot(midwest, 
       aes(x = popdensity, y = percollege, color = state)) +
  geom_point() +
  geom_smooth(se = FALSE)

Your Turn

1. Can you recreate the following figure?

Brief plot customization

ggplot(midwest, 
       aes(x = popdensity, y = percollege, color = state)) +
  geom_point() + 
  scale_x_continuous("Population Density", 
                     breaks = seq(0, 80000, 20000)) + 
  scale_y_continuous("Percent College Graduates") + 
  scale_color_discrete("State")

Brief plot customization Output

Additional ggplot2 resources

• ggplot2 website: http://docs.ggplot2.org/current/index.html
• ggplot2 book: http://www.springer.com/us/book/9780387981413
• R graphics cookbook: http://www.cookbook-r.com/Graphs/

R works as a calculator

1 + 2 - 3

[1] 0

5 * 7

[1] 35

2/1

[1] 2

R Calculator 2

sqrt(4)

[1] 2

2^2

[1] 4

Can save objects to use later

x <- 1 + 3
x

[1] 4

x * 3

[1] 12

R is case sensitive

case_sensitive <- 10
Case_sensitive

Error: object 'Case_sensitive' not found

R Functions

set.seed(1)
rnorm(n = 5, mean = 0, sd = 1)

[1] -0.6264538  0.1836433 -0.8356286  1.5952808  0.3295078

set.seed(1)
rnorm(5, 0, 1)

[1] -0.6264538  0.1836433 -0.8356286  1.5952808  0.3295078

set.seed(1)
rnorm(sd = 1, n = 5, mean = 0)

[1] -0.6264538  0.1836433 -0.8356286  1.5952808  0.3295078

Working through Errors

1. Use ?function_name to explore the details of the function. The examples at the bottom of every R help page can be especially helpful.
   • https://www.rdocumentation.org/ provided by DataCamp is a great alternative as well.
2. If this does not help, copy and paste the error and search on the internet.

Using dplyr for data manipulation

The dplyr package uses verbs for common data manipulation tasks. These include:

• filter()
• count()
• arrange()
• select()
• mutate()
• summarise()

Data

https://fivethirtyeight.com/features/both-republicans-and-democrats-have-an-age-problem/

install.packages('fivethirtyeight')
library(fivethirtyeight)

Using filter

filter(congress_age, congress == 80)

Save filtered results to object

congress_80 <- filter(congress_age, congress == 80)

Other operators for numbers

• >
• <
• >=
• <=

Your Turn

1. Select all rows where the congress member was older than 80 at the start of the term.
2. Use the is.na function to identify congress members that have missing middlenames.

Filter character variables

filter(congress_age, chamber == 'senate')

Combine Operations - AND

filter(congress_age, congress == 80, chamber == 'senate')

Equivalent AND Statement

filter(congress_age, congress == 80 & chamber == 'senate')

Filter - OR

filter(congress_age, congress == 80 | congress == 81)

%in%

filter(congress_age, congress %in% c(80, 81))

Not Operator

filter(congress_age, congress != 80)

Not Operator 2

filter(congress_age, congress == 80 & !chamber == 'senate')

Working with logicals

filter(congress_age, incumbent)

Your Turn

1. Select the Senators from Iowa.
2. Select the Senators from Iowa that are not inbumbents.

All boolean options

Using count

count(congress_age, party)

count(congress_age, incumbent)

Using arrange

arrange(congress_age, state)

Add more variables

arrange(congress_age, state, party)

Descending Order

arrange(congress_age, desc(congress))

Your Turn

1. Count the number of congress members from each party that are older than 80 at term start.
2. Arrange the result from above by the variable n.

Using select

select(congress_age, congress, chamber, party, age)

Helper functions

• starts_with()
• ends_with()
• contains()
• matches()
• num_range()
• :
• everything()

starts_with helper

select(congress_age, starts_with('s'))

Contains helper

select(congress_age, contains('name'))

Colon

select(congress_age, congress:birthday)

Drop variables

select(congress_age, -firstname, -state, -party, -incumbent, -chamber)

Reorder with everything

select(congress_age, congress, chamber, incumbent, age, everything())

rename function

rename(congress_age, first_name = firstname, last_name = lastname)

Your Turn

1. Using the dplyr helper functions, select all the variables that start with the letter ‘c’.
2. Rename the first three variables in the congress data to ‘x1’, ‘x2’, ‘x3’.
3. After renaming the first three variables, use this new data (ensure you saved the previous step to an object) to select these three variables with the num_range function.

Using mutate

congress_red <- select(congress_age, congress, chamber, state, party)
mutate(congress_red, 
       democrat = ifelse(party == 'D', 1, 0),
       num_democrat = sum(democrat)
       )

Your Turn

1. Using the diamonds data, use ?diamonds for more information on the data, use the mutate function to calculate the price per carat. Hint, this operation would involve standardizing the price variable so that all are comparable at 1 carat.
2. Using mutate, calculate the rank of the original price variable and the new price variable calculated above using the min_rank function. Are there differences in the ranking of the prices?

Using summarise

congress_2 <- mutate(congress_age, 
       democrat = ifelse(party == 'D', 1, 0)
       )
summarise(congress_2, 
          num_democrat = sum(democrat)
          )

group_by

congress_grp <- group_by(congress_2, congress)
summarise(congress_grp, 
          num_democrat = sum(democrat),
          total = n(),
          prop_democrat = num_democrat / total
)

Explore trend

num_dem <- summarise(congress_grp, 
                     num_democrat = sum(democrat),
                     total = n(),
                     prop_democrat = num_democrat / total
)
ggplot(num_dem, aes(x = congress, y = prop_democrat)) + 
  geom_line()

Explore trend output

Your Turn

1. Suppose we wanted to calculate the number and proportion of republicans instead of democrats, assuming these are the only two parties, edit the summarise command above to calculate these values.
2. Suppose instead of using sum(democrat) above, we used mean(democrat), what does this value return? Why does it return this value?

group_by with mutate

congress_red <- select(congress_age, congress, chamber, state, party)
congress_grp <- group_by(congress_red, congress)

mutate(congress_grp, 
       democrat = ifelse(party == 'D', 1, 0),
       num_democrat = sum(democrat),
       total = n(),
       prop_democrat = num_democrat / total
)

group_by with mutate output

Chaining operations

summarise(
  group_by(
    mutate(
      filter(
        congress_age, congress >= 100
      ), 
      democrat = ifelse(party == 'D', 1, 0)
    ),
    congress, chamber
  ),
  num_democrat = sum(democrat),
  total = n(),
  prop_democrat = num_democrat / total
)

The pipe %>% is the answer

congress_age %>%
  filter(congress >= 100) %>%
  mutate(democrat = ifelse(party == 'D', 1, 0)) %>%
  group_by(congress, chamber) %>%
  summarise(
    num_democrat = sum(democrat),
    total = n(),
    prop_democrat = num_democrat / total
  )

The two are identical

pipe_congress <- congress_age %>%
  filter(congress >= 100) %>%
  mutate(democrat = ifelse(party == 'D', 1, 0)) %>%
  group_by(congress, chamber) %>%
  summarise(
    num_democrat = sum(democrat),
    total = n(),
    prop_democrat = num_democrat / total
  )
nested_congress <- summarise(
  group_by(
    mutate(
      filter(
        congress_age, congress >= 100
      ), 
      democrat = ifelse(party == 'D', 1, 0)
    ),
    congress, chamber
  ),
  num_democrat = sum(democrat),
  total = n(),
  prop_democrat = num_democrat / total
)
identical(pipe_congress, nested_congress)

[1] TRUE

Your Turn

1. Look at the following nested code and determine what is being done. Then translate this code to use the pipe operator.

summarise(
  group_by(
    mutate(
      filter(
        diamonds, 
        color %in% c('D', 'E', 'F') & cut %in% c('Fair', 'Good', 'Very Good')
      ),
      f_color = ifelse(color == 'F', 1, 0),
      vg_cut = ifelse(cut == 'Very Good', 1, 0)
    ),
    clarity
  ),
  avg = mean(carat),
  sd = sd(carat),
  avg_p = mean(price),
  num = n(),
  summary_f_color = mean(f_color),
  summary_vg_cut = mean(vg_cut)
)

Other Useful dplyr functions

There are a set of functions that can greatly simplify data operations. These functions end with:

• *_if
• *_each
• *_all
• *_at

Simple example

rename_if(congress_age, is.character, toupper)

More realistic example

starwars %>%
  group_by(species, gender) %>%
  summarise_if(is.numeric, mean, na.rm = TRUE)

Can call more than one function

starwars %>%
  group_by(species, gender) %>%
  summarise_if(is.numeric, funs(mean, median), na.rm = TRUE) %>%
  add_tally()

Read in your own data

• We will use data posted to GitHub: https://github.com/lebebr01/iowa_data_science/tree/master/data

Data Import

ufo <- read_csv("https://raw.githubusercontent.com/lebebr01/iowa_data_science/master/data/ufo.csv")

Parsed with column specification:
cols(
  `Date / Time` = col_character(),
  City = col_character(),
  State = col_character(),
  Shape = col_character(),
  Duration = col_character(),
  Summary = col_character(),
  Posted = col_character()
)

Show Data

ufo

Manual column names

ufo_man <- read_csv("https://raw.githubusercontent.com/lebebr01/iowa_data_science/master/data/ufo.csv", 
         skip = 1, 
         col_names = c('Date/Time', 'City', 'State', 
                       'Shape', 'Duration', 'Summary',
                       'Posted'))

Parsed with column specification:
cols(
  `Date/Time` = col_character(),
  City = col_character(),
  State = col_character(),
  Shape = col_character(),
  Duration = col_character(),
  Summary = col_character(),
  Posted = col_character()
)

Manual column names output

ufo_man

Manual column types

ufo_date <- read_csv("Data/ufo.csv", 
         col_types = list(
           'Date / Time' = col_datetime(format = "%m/%d/%y %H:%M"),
           City = col_character(),
           State = col_character(),
           Shape = col_character(),
           Duration = col_character(),
           Summary = col_character(),
           Posted = col_character()
         ))

number of columns of result is not a multiple of vector length (arg 1)56 parsing failures.
row # A tibble: 5 x 5 col     row         col                 expected   actual           file expected   <int>       <chr>                    <chr>    <chr>          <chr> actual 1   119 Date / Time date like %m/%d/%y %H:%M  12/1/14 'Data/ufo.csv' file 2   194 Date / Time date like %m/%d/%y %H:%M 11/27/14 'Data/ufo.csv' row 3   236 Date / Time date like %m/%d/%y %H:%M 11/24/14 'Data/ufo.csv' col 4   407 Date / Time date like %m/%d/%y %H:%M 11/15/14 'Data/ufo.csv' expected 5   665 Date / Time date like %m/%d/%y %H:%M 10/31/14 'Data/ufo.csv'
... ................. ... .................................................................... ........ .................................................................... ...... .................................................................... .... .................................................................... ... .................................................................... ... .................................................................... ........ ....................................................................
See problems(...) for more details.

Manual column types output

ufo_date

Still problems

problems(ufo_date)

Other text formats

• tsv - tab separated files - read_tsv
• fixed width files - read_fwf
• white space generally - read_table
• delimiter generally - read_delim

Your Turn

1. There is a tsv file posted on icon called “lotr_clean.tsv”. Download this and read this data file into R.
2. Instead of specifying the path, use the function file.choose(). For example, read_tsv(file.choose()).
   • What does this function do?
   • Would you recommend this to be used in a reproducible document?

Excel Files

install.packages('readxl')

read_excel

library(readxl)
read_excel('data/titanic.xlsx')

Write Output Files

ufo_count <- ufo %>%
  group_by(State) %>%
  mutate(num_state = n())

write_csv(ufo_count, path = 'path/to/save/file.csv')

Additional Resources

• R for Data Science: http://r4ds.had.co.nz/